Uncertainty

The utility of predictions

1. What determines the limits to the utility of predictions?

The utility of a model/forecast depends on:

• the rate of accumulation of uncertainty (or loss of proficiency) through time, as it tends towards being no better than chance,

combined with

• the precision or “forecast proficiency threshold” required for the decision in question.

Together these determine the “ecological forecast horizon” (Petchey et al. (2015)).

Ecological Forecast Horizon

The ecological forecast horizon (from Petchey et al. (2015)).

Some forecasts may lose proficiency very quickly, crossing (or starting below) the forecast proficiency threshold. If the forecast loses proficiency more slowly, or the proficiency threshold requirements are lower, the forecast horizon is further into the future.

Sources and types of uncertainty

2. What determines prediction uncertainty?

Dietze classifies prediction uncertainty in his book (Dietze 2017a) and subsequent paper (Dietze 2017b) in the form of an equation (note that I’ve spread it over multiple lines):

\[ \underbrace{Var[Y_{t+1}]}_\text{predictive variance} \approx \; \underbrace{stability*uncertainty}_\text{initial conditions} \; + \\ \] \[ \underbrace{sensitivity*uncertainty}_\text{drivers} \; + \\ \] \[ \underbrace{sensitivity*(uncertainty+variability)}_\text{(parameters + random effects)} \; + \\ \] \[ \underbrace{Var[\epsilon]}_\text{process error} \; \; \]

Sources and types of uncertainty

If we break the terms down into (something near) English, we get:

The dependent variable:

\[Var[Y_{t+1}] \approx\]

“The uncertainty in the prediction for the variable of interest (\(Y\)) in the next time step (\(t+1\)) is approximately equal to…”

And now the independent variables (or terms in the model):

Sources of uncertainty: Initial cond.

\[\underbrace{stability*uncertainty}_\text{initial conditions} \; +\]

“The stability multiplied by the uncertainty in the initial conditions, plus”

• Initial conditions = the state of \(Y\) and associated parameters at time \(t_0\) (time of start).
• Stability refers to whether it is a variable with stabilizing feedbacks (think of alternate stable states), versus whether it changes very quickly (with the extreme being one that tends towards chaos such as atmospheric conditions often do).
  • Another example is populations of \(r\) versus \(K\) selected species (i.e. unstable - high growth rate with short-lived individuals that tend to boom and bust versus stable - low growth rate of long-lived individuals with high survival)
• Uncertainty is uncertainty in the state of \(Y\) and parameters due to observation error.

Sources of uncertainty: Drivers

\[\underbrace{sensitivity * uncertainty}_\text{drivers} \; + \]

“The sensitivity to, multiplied by the uncertainty in, external drivers, plus”

• External drivers are just the independent variables (covariates) in the model.
• Predictability of \(Y\) depends on its sensitivity to each covariate (i.e. how much would \(Y\) change for a given change in the covariate), and uncertainty in those covariates.
  • Worst scenario is if \(Y\) is highly sensitive and the covariates are highly uncertain.
  • Since we’re forecasting, uncertainty in each covariate may often reflect how well we can forecast that covariate (e.g. future climate). If we can’t predict \(X\), we can’t use it to predict \(Y\)… (e.g. occurrence of fire in the postfire recovery state space model). That said, if \(Y\) is not very sensitive to \(X\), there’s less of a problem.

Sources of uncertainty: Parameters

\[\underbrace{sensitivity*(uncertainty+variability)}_\text{(parameters + random effects)} + \]

“The sensitivity to, multiplied by uncertainty and variability in, the parameters, plus”

• Parameter uncertainty pertains to how good our estimates of the parameters are.
  • This is usually a question of sample size - “Do we have enough data to obtain a good estimate (i.e. accurate mean, low uncertainty) of the parameters?”.
  • It is also linked to the number of parameters in the model. The more parameters, the more data you need to obtain good parameter estimates. This is another reason to avoid overly complex models.
• Parameter sensitivity is similar to driver sensitivity - “How much change do we expect in \(Y\) for a given change in the parameter?”

Sources of uncertainty: Parameters

\[\underbrace{sensitivity*(uncertainty+variability)}_\text{(parameters + random effects)} + \]

“The sensitivity to, multiplied by uncertainty and variability in, the parameters, plus”

• The overall contribution of a parameter to the predictive variance (i.e. uncertainty in the forecast) depends on its sensitivity multiplied by its uncertainty.
  • Targeting effort (fieldwork etc) to better constrain poorly estimated parameters is one of the best ways to reduce prediction uncertainty.
• Parameter variability reflects factors that cause deviation (or offsets) from the mean of the parameter that may be known, but may either be poorly estimated or not included in the rest of the model.
  • These are random effects that can be caused by factors that create autocorrelation like space, time, phylogeny, etc.

Sources of uncertainty: Process error

\[\underbrace{Var[\epsilon]}_\text{process error}\] “The process error.”

• This refers to errors in the model due to structural uncertainty and stochasticity.
• Stochasticity refers to ecological phenomena of relevance that are very difficult to predict (at least within the context of the focal model).
  • Examples include the occurrence of fire, dispersal or mortality that are often chance events similar to a coin toss.
• Model structural uncertainty simply reflects that all models are simplifications of reality and none are perfect. We’ll always be missing something.
  • That said, this could also include “user error” where the forecaster specified the wrong process model, or applied the wrong probability distribution in the data model, etc.
  • Working with multiple models and employing model selection or averaging can help reduce structural uncertainty (in addition to thinking really hard about how to specify a better model of course…).

Propagating uncertainty

3. “How can we propagate uncertainty through our models and into our predictions?”

There are many methods, but it’s worth recognizing that these are actually two steps:

1. Propagating uncertainty through the model
   • i.e. in fitting the model, so we can include uncertainty in our parameter estimates
   • This is typically focused on “How does the uncertainty in X affect the uncertainty in Y?”
2. Propagating uncertainty into our forecasts
   • i.e. exploring the implications of uncertainty in our model (parameters etc) for our confidence in the forecast when making predictions with our fitted model
   • Here we focus on “How do we forecast Y with uncertainty?”
   • This second step is actually the first step in data assimilation, which is the subject of the next section/lecture

Propagating uncertainty

This could be a lecture series of its own. In short, there are 5 methods to address step 1, and most have related methods for step 2 (see Table in notes). The methods differ in whether they:

• Return distributions (e.g. Gaussian curve) or moments (means, medians, standard deviations, etc)
• They have analytical solutions, or need to be approximated numerically

They also have trade-offs between efficiency vs flexibility. The most efficient have the most rigid requirements and assumptions (analytical), while the most flexible (numeric) can be computationally taxing (or impossible given a complex enough model).

Propagating uncertainty

Note: It is possible to propagate uncertainty through the model and into your forecast in one step with Bayesian methods, by treating the forecast states as “missing data” values and estimating posterior distributions for them. This would essentially fit with Monte Carlo methods in the table. This approach may not suit all forecasting circumstances though.

Analyzing and reducing uncertainty

4. “How can we reduce prediction uncertainty?

Working out where it’s coming from (by analyzing and partitioning the sources of uncertainty).

Targeting sources of uncertainty that can be reduced with the best return on investment (important to note that these may not be the biggest sources of uncertainty, just the cheapest and easiest to resolve).

Analyzing and reducing uncertainty

Addressing 1 requires looking at the two ways in which things can be important for the uncertainty in predictions (largely covered in Dietze’s equation above):

• because they’re highly uncertain, which requires you to:
  • propagate uncertainty through the model as above
  • partition uncertainty among your different drivers (covariates) and parameters
• because they’re highly sensitive, requiring you to perform:
  • sensitivity analysis
    • You’ll probably cover these in more detail in Res’ module when looking at matrix models. I’m not going to go into them here, but they focus on how a change in X translates into a change in Y. The bigger the relative change in Y, the more sensitive.

Analyzing and reducing uncertainty

Addressing 2 may not be as straightforward as you’d hope. Parameters that are highly uncertain and to which your state variable (Y) are highly sensitive will cause the most uncertainty in your predictions. That said, given limited resources, they may not be the best target for reducing uncertainty for a number of reasons, e.g.

• they may be inherently uncertain and uncertainty may remain high even with vast sampling effort
  • power analysis can help here (i.e. exploring how uncertainty changes with sample size)
• they may be hugely costly or time-consuming, trading off against resources you could focus on reducing other sources of uncertainty

In fact, by this stage you should have most of the pieces of the puzzle to help you build a model to predict where your effort is best invested by exploring the relationship between sample size and variance contribution to overall model uncertainty! You can even include economic principles to estimate monetary or person-hour implications. This is called observational design.

Invasive alien plants and streamflow

A reminder of Bayes Theorem

\[ \underbrace{p(\theta|D)}_\text{posterior} \; \propto \; \underbrace{p(D|\theta)}_\text{likelihood} \;\; \underbrace{p(\theta)}_\text{prior} \; \]

The posterior

The posterior is the conditional probability of the parameters given the data \(p(\theta|D)\) and provides a probability distribution for the values any parameter can take,

This allows us to represent uncertainty in the model and forecasts as probabilities, which is powerful for indicating the probability of our forecast being correct.

A reminder of Bayes Theorem

\[ \underbrace{p(\theta|D)}_\text{posterior} \; \propto \; \underbrace{p(D|\theta)}_\text{likelihood} \;\; \underbrace{p(\theta)}_\text{prior} \; \]

The likelihood

The likelihood \(p(D|\theta)\) represents the probability of the data \(D\) given the model with parameter values \(\theta\), and is used in analyses to find the likelihood profiles of the parameters.

This term looks for the best estimate of the parameters using Maximum Likelihood Estimation, where the likelihood of the parameters are maximized for a given model by choosing the parameters that maximize the probability of the data.

A reminder of Bayes Theorem

\[ \underbrace{p(\theta|D)}_\text{posterior} \; \propto \; \underbrace{p(D|\theta)}_\text{likelihood} \;\; \underbrace{p(\theta)}_\text{prior} \; \]

The prior

The prior is the marginal probability of the parameters, \(p(\theta)\).

It represents the credibility of the parameter values, \(\theta\), without the data, and is specified using our prior belief of what the parameters should be, before interrogating the data. This provides a formal probabilistic framework for the scientific method, in that new evidence must be considered in the context of previous knowledge, providing the opportunity to update our beliefs.

Advantages of Bayesian approaches

Data fusion

Least Squares

Least Squares makes no distinction between the process model and the data model.

Least Squares

Here I’ll use the example used in the practical, exploring time-series of the Normalised Difference Vegetation Index (NDVI ~ vegetation “greenness” measured from the MODIS satellites) to look at vegetation recovery with time since fire in Fynbos.

Least Squares

Here I use Nonlinear Least Squares (NLS) to fit a negative exponential model:

\[\begin{gather} \text{NDVI}_{i,t}=\alpha_i+\gamma_i\Big(1-e^{-\frac{age_{i,t}}{\lambda_i}}\Big) \end{gather}\]

This process model is the only opportunity for inputting data, and we input time series for \(NDVI\) and date (or postfire \(age\), since our time series started at the time of a fire). Parameters \(\alpha\), \(\gamma\) and \(\lambda\) are estimated by the model.

Least Squares

The only way to add other data sources is to make the process model more complex.

• E.g to capture the seasonal fluctuations in NDVI we can add a sine term that requires us to specify the month (\(m\)) of the fire, like so:

\[\begin{gather} \text{NDVI}_{i,t}=\alpha_i+\gamma_i\Big(1-e^{-\frac{age_{i,t}}{\lambda_i}}\Big)+ A_i\text{sin}\Big(2\pi\times\text{age}_{i,t}+\Big[\phi+\frac{\pi}{6}(m_{i,t}-1)\Big]\Big) \end{gather}\]

A quick aside on the parameters

\[\begin{gather} \text{NDVI}_{i,t}=\alpha_i+\gamma_i\Big(1-e^{-\frac{age_{i,t}}{\lambda_i}}\Big)+ A_i\text{sin}\Big(2\pi\times\text{age}_{i,t}+\Big[\phi+\frac{\pi}{6}(m_{i,t}-1)\Big]\Big) \end{gather}\]

Maximum Likelihood

Maximum Likelihood makes a distinction between the data model and the process model:

Maximum Likelihood

I don’t make the distinction in the equations presented in the practical, but do in the functions, where we included the data model, where we specified a Gaussian (normal) distribution around the mean, \(\mu\) (which is described by the process model):

\[\begin{gather} NDVI_{i,t}\sim\mathcal{N}(\mu_{i,t},\frac{1}{\sqrt{\tau}}) \\ \end{gather}\]

Where \(\mathcal{N}(\mu_{i,t},\frac{1}{\sqrt{\tau}})\) is just a normal distribution with mean \(\mu\) and variance \(\frac{1}{\sqrt{\tau}}\).

Maximum Likelihood

We then have a separate function for the process model, which describes \(\mu\) as a function of the covariate (vegetation \(age\)), with parameters \(\alpha\), \(\gamma\), \(\lambda\), \(A\), \(\phi\) and \(m\):

\[\begin{gather} \mu_{i,t}=\alpha_i+\gamma_i\Big(1-e^{-\frac{age_{i,t}}{\lambda_i}}\Big)+ A_i\text{sin}\Big(2\pi\times\text{age}_{i,t}+\Big[\phi+\frac{\pi}{6}(m_{i,t}-1)\Big]\Big) \end{gather}\]

Maximum Likelihood

The beauty of a separate data model is that you have flexibility to specify probability distributions that suit the data observation process (i.e. not just the normal distribution), e.g. Binomial coin flips, Poisson counts of individuals, Exponential waiting times, etc.

You can even specify custom data models with their own covariates, which is useful if you have information on things like instrument drift and calibration, etc - i.e. another opportunity for data fusion.

Single-level Bayes

When we use Bayesian models we now have the priors, which are essentially models describing our prior expectation for each of the parameters in the process model, like so:

Bayesian models include parameter models that describe our prior expectation for the parameters in the process.

Single-level Bayes

When implementing Bayesian models, we specify the priors as parameter models, e.g.:

\[\begin{gather} \alpha_i\sim\mathcal{N}(\mu_{\alpha},\frac{1}{\sqrt{\tau_{\alpha}}})\\ \gamma_i\sim\mathcal{N}(\mu_{\gamma},\frac{1}{\sqrt{\tau_{\gamma}}})\\ \lambda_i\sim\mathcal{N}(\mu_{\lambda},\frac{1}{\sqrt{\tau_{\lambda}}})\\ \end{gather}\]

Where (in this case) we’re saying we believe the three parameters from our simpler postfire recovery model are all sampled from normal distributions with independent¹ means and variances.

Single-level Bayes

Note that you need priors (i.e. parameter models) for all parameters. They don’t all have to be independent though, which can be useful, for example if you have multiple separate sets drawn from the same population, etc.

Single-level Bayes

When combining Multiple Likelihoods, there need to be links between all terms - e.g. two datasets (\(D_1\) and \(D_2\)) that share the same parameters (\(\theta\)):

\[ \underbrace{p(D_1|\theta)}_\text{likelihood 1} \;\; \underbrace{p(D_2|\theta)}_\text{likelihood 2} \;\;\underbrace{p(\theta)}_\text{prior} \; \]

or one dataset (\(D_1\)) conditional on another (\(D_2\)), that is conditional on parameters (\(\theta\)):

\[ \underbrace{p(D_1|D_2)}_\text{likelihood 1} \;\; \underbrace{p(D_2|\theta)}_\text{likelihood 2} \;\;\underbrace{p(\theta)}_\text{prior} \; \] etc.

Hierarchical Bayes

Hierarchical Bayes is a form of multilevel modelling that provides incredible flexibility for model specification through specifying the priors as models with inputs and specifying relationships among the priors or from one prior to multiple layers in the model.

• i.e. Hierarchical Bayesian models allow you to fuse data through the parameter model, but when you start treating your priors as parameters, you have to specify priors on your priors (hyperpriors) to your parameter model.

Hierarchical Bayesian models allow considerable flexibility through the inclusion of hyperparameters that can drive the priors.

Hierarchical Bayes

I won’t explore all the options (because they’re almost endless). In the example below, we used environmental covariates (soil, climate, topography, etc.) to explain the variation in the priors and constrain the parameters of the postfire recovery curve.

Until now we have only been considering the postfire recovery curve for a single location, but one can fit every MODIS satellite pixel in the Fynbos Biome in one model, like Wilson, Latimer, and Silander (2015) did, (including the seasonality term in the process model).

Hierarchical Bayes

Hierarchical Bayes

Hierarchical Bayes

Advantages of including the regression of the parameters on environmental covariates in this model include:

Data fusion, model complexity and uncertainty

Increasing model complexity provides opportunities for fusing new data into your model, but be aware that this comes with trade-offs.

Utility of the postfire model

The model does not make explicit near-term forecasts, but gives estimates of the expected NDVI signal for any location with known age since fire and time of year.

We’re using this to develop a near-real time satellite change detection system for Fynbos.

The Ecosystem Monitoring for Management Application (EMMA). See www.emma.eco for more.

EMMA workflow

EMMA

Managers can get an overview of change in the landscape every time new MODIS satellite data are collected (daily, or 16-day averages).

Overview map highlighting some of the major changes detected by Slingsby, Moncrieff, and Wilson (2020).